Completing functional testing

ABSTRACT

The Functional Tester Plug-in (FTP) extracts information from a data base to enhance automated testing tools to accommodate changes to the computer environment and so that a simulation can be fully executed even if an exception occurs. The FTP accesses the test script and static data store created by a developer using the automated testing tool, wherein the test script and static data store indicate a first resource to be tested at a verification point and a first expected result. The FTP identifies the first resource used by the test script at a first verification point and accesses the data base. The FTP identifies the status and configuration of the first resource in the data base. The FTP calculates an optimal expected result based on any changes in the status and configuration of the first resource.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.13/416,160, filed on Mar. 9, 2012, which a Continuation of U.S.application Ser. No. 11/744,268, filed on May 4, 2007.

FIELD OF THE INVENTION

The present invention relates generally to data processing and relatesspecifically to testing and simulating performance of software.

BACKGROUND OF THE INVENTION

When developers create a new computerized application, the newapplication is tested to ensure it functions properly. Automated testingtools are used to run simulations on an application. The results of thesimulations indicate whether the application works properly under avariety of predefined conditions. One example of an automated testingtool is IBM's Rational Functional Tester. Although an application willbe used to describe this invention, automated testing tools can also beused to test Graphical User Interface (GUI) designs and GUI flows.Developers use the results of the simulations to ensure that theapplication or the GUI will work properly in the future.

The first step to running a simulation is to record a static test. Thedeveloper performs a series of manipulations using the application, asif the developer was a user. The developer defines limits and parametersbased on expected user inputs. The recording of the static test iscalled a “test script.” The developer also sets verification points forthe simulation by specifying expected results from a step or series ofsteps in the test script. The expected results are saved to a “staticdata store” which also contains data indicating any externalapplications used by the application when executing the test script. Alimitation to using a static test script is that if the computerenvironment changes after the script is written, the simulation may notreflect the new computer environment. It would be desirable to have anautomatic way of updating the static test script to reflect changes inthe computer environment at the time the simulation is run.

The second step to running the simulation is executing the series ofrecorded manipulations from the test script. At each verification point,the functional tester compares the actual results to the expectedresults in the static data store. When the actual results do not matchthe expected results the test fails and the simulation ends. When a testfails, it is called an “exception.”

If the simulation ends when an exception occurs, the developer only hasenough information to correct the single problem causing the exception.The developer must resolve the issue that caused the exception and runthe simulation again before subsequent verification points can betested. It would be desirable if the automated testing tool couldcontinue the simulation after an exception occurs and identify if therest of the test script works properly or if there are other problems.

One type of exception that occurs when running simulations is theinability to properly access a computer resource needed by theapplication when running the test script. This type of exception is notcaused by bad coding, but by changes in the computer environment such asthe computer resource having been moved, renamed, replaced, updated, orreconfigured. Information related to the configuration of and therelationships between computer resources on the computer environment isoften stored in a data base, such as a Configuration Management DataBase (CMDB). Information related to each resource stored in the CMBD iscalled a “Configuration Item” (CI).

A need exists for a method of using a data base, such as a CMDB, toenhance software testing simulations by ensuring the simulation can beexecuted to the fullest extent. The test script can be updated atrun-time using the data base to accommodate changes in the computerenvironment. If an exception occurs at a simulation verification point,the automated testing tool can identify a related resource, as indicatedin the data base, to repeat the simulation verification point andcomplete the simulation.

SUMMARY OF THE INVENTION

The Functional Tester Plug-in (FTP) extracts information from a database to enhance automated testing tools to accommodate changes to thecomputer environment and so that a simulation can be fully executed evenif an exception occurs. The FTP accesses the test script and static datastore created by a developer using the automated testing tool, whereinthe test script and static data store indicate a first resource to betested at a verification point and a first expected result. The FTPidentifies the first resource used by the test script at a firstverification point and accesses the data base. The FTP identifies thestatus and configuration of the first resource in the data base. The FTPcalculates an optimal expected result based on any changes in the statusand configuration of the first resource. The FTP executes the testscript using the optimal expected result based on any changes in thestatus and configuration of the first resource. Additionally, the FTPidentifies a second resource related to the first resource from a recordin the data base and calculates a second expected result based onexecuting the test script with the second resource. Responsive to anexception generated by the automated testing tool at the firstverification point, the FTP repeats the test at the first verificationpoint using the second resource. In an alternate embodiment, the FTPuses the data base to generate a set of optimal resources and a set ofexpected results to test at a verification point.

BRIEF DESCRIPTION OF DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives and advantages thereof, willbe understood best by reference to the following detailed description ofan illustrative embodiment when read in conjunction with theaccompanying drawings, wherein:

FIG. 1 is an exemplary computer network;

FIG. 2 describes programs and files in a memory on a computer;

FIG. 3 is a flowchart of an analysis component;

FIGS. 4A and 4B are an exemplary set of meta-data definitions used bythe analysis component; and

FIG. 5 is a flowchart of an exception component.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The principles of the present invention are applicable to a variety ofcomputer hardware and software configurations. The term “computerhardware” or “hardware,” as used herein, refers to any machine orapparatus that is capable of accepting, performing logic operations on,storing, or displaying data, and includes without limitation processorsand memory. The term “computer software” or “software,” refers to anyset of instructions operable to cause computer hardware to perform anoperation. A “computer,” as that term is used herein, includes withoutlimitation any useful combination of hardware and software, and a“computer program” or “program” includes without limitation any softwareoperable to cause computer hardware to accept, perform logic operationson, store, or display data. A computer program may, and often is,comprised of a plurality of smaller programming units, including withoutlimitation subroutines, modules, functions, methods, and procedures.Thus, the functions of the present invention may be distributed among aplurality of computers and computer programs. The invention is describedbest, though, as a single computer program that configures and enablesone or more general-purpose computers to implement the novel aspects ofthe invention. For illustrative purposes, the inventive computer programwill be referred to as the “Functional Tester Plug-in” or “FTP.”

Additionally, the FTP is described below with reference to an exemplarynetwork of hardware devices, as depicted in FIG. 1. A “network”comprises any number of hardware devices coupled to and in communicationwith each other through a communications medium, such as the Internet. A“communications medium” includes without limitation any physical,optical, electromagnetic, or other medium through which hardware orsoftware can transmit data. For descriptive purposes, exemplary network100 has only a limited number of nodes, including workstation computer105, workstation computer 110, server computer 115, and persistentstorage 120. Network connection 125 comprises all hardware, software,and communications media necessary to enable communication betweennetwork nodes 105-120. Unless otherwise indicated in context below, allnetwork nodes use publicly available protocols or messaging services tocommunicate with each other through network connection 125.

FTP 200 typically is stored in a memory, represented schematically asmemory 220 in FIG. 2. The term “memory,” as used herein, includeswithout limitation any volatile or persistent medium, such as anelectrical circuit, magnetic disk, or optical disk, in which a computercan store data or software for any duration. A single memory mayencompass and be distributed across a plurality of media. Further FTP200 may reside in more than one memory distributed across differentcomputers, servers, logical partitions, or other hardware devices. Theelements depicted in memory 220 may be located in or distributed acrossseparate memories in any combination, and FTP 200 may be adapted toidentify, locate and access any of the elements and coordinate actions,if any, by the distributed elements. Thus, FIG. 2 is included merely asa descriptive expedient and does not necessarily reflect any particularphysical embodiment of memory 220. As depicted in FIG. 2, though, memory220 may include additional data and programs. Of particular import toFTP 200, memory 220 may include Functional Tester 230, test script 240,meta-data definitions 245, static data store 250, analysis store 260,CMDB 270, and new application 280 with which FTP 200 interacts.

Functional Tester 230 is a prior art simulation tool, such as IBM'sRational Functional Tester, and has two prior art components related tothe present invention: Setup 232 and Execute 234. Setup 232 is used by adeveloper to create test script 240 and meta-data definitions 245 basedon new application 280. Setup 232 also allows the developer to createstatic data store 250 indicating verification points, other resourcesneeded to execute test script 240, and expected results for eachverification point. Execute 234 runs test script 240, comparing expectedresults in from static data store 250 at each verification pointdesignated by the developer.

CMDB 270 is a prior art Configuration Management Database, shown in FIG.2 having configuration item A 272, configuration item B 274, andconfiguration item C 276 corresponding to resource A 972, resource B974, and resource C 976 respectively. CMDB 270 is a repository of allthe resources on computer network 100. Configuration items A 272, B 274,and C 276 are records in CMDB 270 containing relationships betweendifferent resources, versions of the same resource, and theconfiguration of the resources. Resources may be any resource on network100, such as files, hardware, firmware, software, and documentation.Although this example shows CMBD 270, FTP 200 could use any data base orother data source containing the configuration of and relationshipsbetween resources on network 100.

FTP 200 has two components: Analysis Component 300 and ExceptionComponent 400. Analysis Component 300 creates analysis store 260 basedon information in CMDB 270 whenever execute 234 of functional tester 230runs a simulation of new application 280. Analysis store 260 describesan optimized set of resources and expected verification point resultsbased on the run-time configuration of resources on network 100.Whenever an exception occurs at a verification point, ExceptionComponent 400 causes execute 234 to repeat the failed verification pointusing a related resource from analysis store 260.

Analysis Component 300 is shown in FIG. 3. Analysis Component 300 starts(310) whenever execute 234 of functional tester 230 runs a simulation onnew application 280. Analysis Component 300 accesses test script 240,meta-data definitions 245, and static data store 250 (312) thenidentifies the first verification point (314). If a resource, such asresource A 292, on network 100 is needed by test script 240 at theverification point (316), Analysis Component 300 accesses CMDB 270(318). Analysis Component 300 uses meta-data definitions 245 to analyzeconfiguration item A 272, corresponding with resource A 292, todetermine the configuration of resource A 292, identify anyconfiguration changes of resource A 292, and identify any relationshipsto another resource, such as resource B 294 (320). Analysis Component300 calculates an optimal set of resources needed by the verificationpoint and determines expected results for any related resources (322).Analysis Component 300 saves the optimal set of resources and expectedresults to analysis store 260 (324). After saving to analysis store 260,or if the first verification point did not require a resource on network100 at step 316, Analysis Component 300 determines if there is anotherverification point in test script 240 and data store 250 (326). If thereis another verification point at step 326, Analysis Component 300repeats steps 316-326 until all verification points have been analyzed.Once all verification points in test script 240 and data store 250 havebeen analyzed, Analysis Component 300 stops (330).

Meta-data definitions 245 contains a series of queries in XML formatthat can be used by analysis component 300 when determining an optimalset of resources needed to test a verification point. FIG. 4 shows anexemplary XML listing of definitions used when querying CMBD 270 andcreating the optimal set of resources to use at a verification point atsteps 318-322 above.

Exception Component 400 is shown in FIG. 5. Exception Component 400starts when a failure occurs at a verification point when execute 234 offunctional tester 230 runs a simulation on new application 280 (410).For example, a verification point in test script 240 requiring use ofresource A 292 fails because resource A 292 is not available. ExceptionComponent 400 pauses execute 234 (412) and opens analysis store 260(414). Exception Component 400 selects a related resource and anexpected result from using the related resource (416). For example,Exception Component 400 may select resource B 294 which is capable ofperforming the same task as resource A 292. Exception Component 400restarts execute 234, repeating the test script at the failedverification point, but using the related resource, such as resource B294 (418) then stops (420). Because Exception Component 400 responds toevery failed verification point, it repeats the selection of relatedresources as necessary until execute 234 completes test script 240.

In an alternate embodiment, not shown in the figures, Analysis Component300 is adapted to create multiple test scripts so that the simulationcan be run multiple times for different sets of resources. For example,if new application 280 uses resource A 292 at the first verificationpoint in test script 240. Analysis Component 300 may discover whenquerying CMDB 270 using meta-data definitions 245 that resource B 294and resource C 296 are related to resource A 292. Analysis Component 300would create a first alternate test script to repeat the simulationusing resource B 294, and a second alternate test script to repeat thesimulation using resource C 296. As with the primary embodiment,Analysis Component 300 will calculate expected results for the alternatetest scripts and place the results in the proper data store forverification purposes. This embodiment allows the developer to morefully test new application 280 without requiring extra effort to createmultiple test scripts.

A preferred form of the invention has been shown in the drawings anddescribed above, but variations in the preferred form will be apparentto those skilled in the art. The preceding description is forillustration purposes only, and the invention should not be construed aslimited to the specific form shown and described. The scope of theinvention should be limited only by the language of the followingclaims.

What is claimed is:
 1. A computer-implemented process for enhancing anautomated testing tool, comprising: accessing a test script and a datastore, the test script and the static data store indicating a firstresource to be tested at a verification point and the data storecontains a first expected result for the verification point; accessing adatabase including configuration information related to the firstresource; identifying, from the database, a configuration change of thefirst resource; calculating an updated first expected result based uponthe configuration change; executing the test script at the verificationpoint using the updated first expected result; identifying an optimumset of resources in the database related to the first resource;calculating an optimum set of expected results based on a configurationof the optimum set of resources; and executing the test script at theverification point using the optimum set of resources.
 2. The process ofclaim 1, wherein the automated testing tool is the Rational FunctionalTester.
 3. The process of claim 1, wherein the database is aConfiguration Management Database (CMDB).
 4. The process of claim 1,wherein metadata definitions are used to query the database with respectto the first resource.
 5. A computer hardware system configured toenhance an automated testing tool, comprising: at least one hardwareprocessor, wherein the at least one hardware processor is configured toinitiate and/or perform: accessing a test script and a data store, thetest script and the static data store indicating a first resource to betested at a verification point and the data store contains a firstexpected result for the verification point; accessing a databaseincluding configuration information related to the first resource;identifying, from the database, a configuration change of the firstresource; calculating an updated first expected result based upon theconfiguration change; executing the test script at the verificationpoint using the updated first expected result; identifying an optimumset of resources in the database related to the first resource;calculating an optimum set of expected results based on a configurationof the optimum set of resources; and executing the test script at theverification point using the optimum set of resources.
 6. The system ofclaim 5, wherein the automated testing tool is the Rational FunctionalTester.
 7. The system of claim 5, wherein the database is aConfiguration Management Database (CMDB).
 8. The system of claim 5,wherein meta-data definitions are used to query the database withrespect to the first resource.
 9. A computer program product,comprising: a computer memory having stored therein computer usableprogram instructions for enhancing an automated testing tool, thecomputer usable program instructions, which when executed by a computerhardware system, cause the computer hardware system to perform:accessing a test script and a data store, the test script and the staticdata store indicating a first resource to be tested at a verificationpoint and the data store contains a first expected result for theverification point; accessing a database including configurationinformation related to the first resource; identifying, from thedatabase, a configuration change of the first resource; calculating anupdated first expected result based upon the configuration change;executing the test script at the verification point using the updatedfirst expected result; identifying an optimum set of resources in thedatabase related to the first resource; calculating an optimum set ofexpected results based on a configuration of the optimum set ofresources; and executing the test script at the verification point usingthe optimum set of resources.
 10. The computer program product of claim9, wherein the automated testing tool is the Rational Functional Tester.11. The computer program product of claim 9, wherein the database is aConfiguration Management Database (CMDB).
 12. The computer programproduct of claim 9, wherein meta-data definitions are used to query thedatabase with respect to the first resource.